Data-intensive Computing Overview

Principles

Definition

Data-intensive computing is an academic concept and methodological approach concerned with the computational systems and techniques required to effectively store, manage, process, and analyze datasets whose scale and complexity exceed the capabilities of traditional computing paradigms. This field investigates the challenges of harnessing vast data volumes to extract valuable insights, focusing on issues such as scalability, performance, fault tolerance, and data governance in large-scale distributed environments.

Ontological type

Core Architectures

Programming Models

Storage and Data Management

Key developments

Key Figures

Data Locality Foundations era

Ian Foster [1] is a prominent figure in the data locality foundations of this era, with affiliations including Northwestern University [3] and the University of Chicago [4]. His key contributions in this era include Chimera: a virtual data system for representing, querying, and automating data derivation [5], Decoupling computation and data scheduling in distributed data-intensive applications [6], and Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing [7], which collectively reduced data movement and supported scalable analysis by representing data derivation, decoupling computation from data locality, and improving transport and replica management. Michael Wilde [2] is a key contributor in this era, connected with the University of Chicago [4]. His notable contribution is Chimera [5], a virtual data system for representing, querying, and automating data derivation, which illustrated the feasibility of data-centric workflows in data-intensive computing.

Programmable DAG Analytics era

Joseph E. Gonzalez [1] conducted research across University of California, Berkeley [3] and Carnegie Mellon University [4] during the Programmable DAG Analytics era. Gonzalez's key contributions include Apache Spark [6] and Distributed GraphLab [7], which advanced data-centric, DAG-driven analytics by enabling scalable in-memory processing and graph-parallel computation, essential for iterative workloads in this era. Ion Stoica [2] contributed from Johns Hopkins University [5] and University of California, Berkeley [3] during the same era. Stoica's contributions in this era include Apache Spark [6] and Scaling spark in the real world [8], illustrating how Spark's platform matured to support petabyte-scale workloads on commodity clusters and clouds.

Memory-Centric Optimization era

Onur Mutlu [1] is associated with King Abdullah University of Science and Technology [3] and École Polytechnique Fédérale de Lausanne [4] in the Memory-Centric Optimization era. His key contributions in this era include Ambit [7], a memory-centric analytics framework, and Efficient document analytics on compressed data [8], which together reduce data movement and I/O bottlenecks. Phillip B. Gibbons [2] is affiliated with Carnegie Mellon University [5] and ETH Zurich [6] in this era. His key contributions include Ambit [7] and Efficient document analytics on compressed data [8], advancing memory-centric optimization by enabling analytics on compressed data and reducing data movement, which were crucial for scaling large workloads.